Skip to content

gh-158140: Resolve lazily imported sibling submodules independently - #158521

Open
pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:fix-158140-independent-submodules
Open

pablogsal wants to merge 1 commit into
python:mainfrom
pablogsal:fix-158140-independent-submodules

Conversation

@pablogsal

@pablogsal pablogsal commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Fixes #158140. With lazy import pkg.a followed by lazy import pkg.b, accessing pkg.a currently imports pkg.b first. Resolve ordinary dotted imports through the existing pending-child lookup, without publishing placeholders in package dictionaries. Custom namespace builtins and aliased parents retain their existing import behavior.

The lazy-import, import, importlib, module, sys and C API suites pass in debug and free-threaded builds. Regression tests cover independent siblings, star imports, declaration cleanup, deleted attributes, import hooks and failed-import retries.

@pablogsal

Copy link
Copy Markdown
Member Author

Sorry fucked up the squash give me sec

@pablogsal
pablogsal force-pushed the fix-158140-independent-submodules branch from 56babbc to 4e83bfa Compare October 1, 2026 19:23
@Yhg1s

Yhg1s commented Oct 1, 2026

Copy link
Copy Markdown
Member

Is this ready now @pablogsal?

@Yhg1s

Yhg1s commented Oct 2, 2026

Copy link
Copy Markdown
Member

Here's a somewhat surprising change in behaviour: given these two modules:

A.py:

lazy import xml.missing # CORRECT ONE
import B

xml.missing

B.py:

lazy import xml.missing # WRONG ONE

Before this PR the use of xml.missing in A.py will print a traceback showing the lazy import in A.py as the cause:

% base/python A.py
Traceback (most recent call last):
  File "A.py", line 1, in <module>
    lazy import xml.missing # CORRECT ONE
ImportError: lazy import of 'xml.missing' raised an exception during resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "A.py", line 4, in <module>
    xml.missing
    ^^^
ModuleNotFoundError: No module named 'xml.missing'

With this PR, it reports the lazy import in B.py:

% pr/python A.py
Traceback (most recent call last):
  File "B.py", line 1, in <module>
    lazy import xml.missing # WRONG ONE
ImportError: lazy import of 'xml.missing' raised an exception during resolution

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "A.py", line 4, in <module>
    xml.missing
ModuleNotFoundError: No module named 'xml.missing'

Swapping the order of the lazy imports used to not have an effect on the traceback, but now it does: the last lazy import wins. I'm not sure if that's fixable, but I think if we need a single location the first lazy import would be a better winner.

@Yhg1s

Yhg1s commented Oct 2, 2026

Copy link
Copy Markdown
Member

Swapping the order of the lazy imports used to not have an effect on the traceback, but now it does: the last lazy import wins. I'm not sure if that's fixable, but I think if we need a single location the first lazy import would be a better winner.

Okay I think I see why we can't do that, either. I think this is fixable later (perhaps by merging lazy objects, storing all the locations and selecting an appropriate one when printing the traceback). Still, it's a little annoying that this used to work as users would expect.

@Yhg1s

Yhg1s commented Oct 2, 2026

Copy link
Copy Markdown
Member

I think there's a similar issue with modules defining __import__, although I don't have a simple reproducer: if you have two different modules lazily importing the same pkg.module, with their own __import__ methods shadowing the builtin __import__, module A may end up calling module B's __import__ when it does its reification. I can't see how to avoid that one without giving A and B its own sys.modules entirely.

_PyLazyImport_IsActive(PyObject *declaration)
{
return FT_ATOMIC_LOAD_INT_RELAXED(
((PyLazyImportObject *)declaration)->lz_active);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
((PyLazyImportObject *)declaration)->lz_active);
PyLazyImportObject_CAST(declaration)->lz_active);

int
_PyLazyImport_IsActive(PyObject *declaration)
{
return FT_ATOMIC_LOAD_INT_RELAXED(

@Yhg1s Yhg1s Oct 2, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could probably do with an assert.

Suggested change
return FT_ATOMIC_LOAD_INT_RELAXED(
assert(PyLazyImport_CheckExact(declaration));
return FT_ATOMIC_LOAD_INT_RELAXED(

@Yhg1s Yhg1s left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All things considered, LGTM. I think the semantic changes are acceptable for 3.15 and we can see if we can improve the error reporting later.

if (lz == NULL) {
return NULL;
}
PyLazyImportObject *child = (PyLazyImportObject *)lz;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PyLazyImportObject *child = (PyLazyImportObject *)lz;
PyLazyImportObject *child = PyLazyImportObject_CAST(lz);

_PyLazyImport_LoadChild(PyThreadState *tstate, PyObject *declaration,
PyObject *name)
{
PyLazyImportObject *source = (PyLazyImportObject *)declaration;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PyLazyImportObject *source = (PyLazyImportObject *)declaration;
assert(PyLazyImport_CheckExact(declaration));
PyLazyImportObject *source = PyLazyImportObject_CAST(declaration);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

Accessing one lazily imported submodule (lazy import pkg.a) also imports a later imported pkg.b

3 participants